Background
The number of online surveys in German official statistics has been steadily growing over the years. So far, the CAWI-mode is mainly implemented in business and administrative statistics, but there is also a huge demand for online questionnaires in social statistics. Reasons are manifold: E.g. restrictions of financing and time, also expectations of the population to respond to questionnaires online, play an important role.
Hence, to guarantee high quality of statistical data, online questionnaires need to be easy to handle for respondents. Consequently a sophisticated and thoughtful programming is needed. However, only building functionality into a product doesn’t guarantee that people will be able to use it. Therefore the Federal Statistical Office implements usability-tests with online questionnaires to include the view of respondents into consideration.
Methodology
In usability-tests in the Federal Statistical Office a three step approach is applied: Firstly, we observe eye movements and facial expressions, while respondents deal with the questionnaire. Secondly, we conduct cognitive interviews, in which we show the video of eye movements during the fill-in process to the probands. By using cognitive methods we try to figure out reasons for incomplete or missing answers. In a third step, we analyse and interpret the eye tracking data. As there are benefits from every method on the one hand, each method has its weaknesses, too. Therefore we combine information from the three sources with each other (“triangulation”), to provide higher data quality in terms of more objective results and a richer overall picture.
Results
Based on the results of four usability-tests and the findings from current research, the Federal Statistical Office develops a profile of requirements for online questionnaire design in social statistics. The ongoing project addresses items such as the general format of the questionnaire, to design und functionality of navigation, explanations, completion instructions, error messages, question and answering formats and accessibility.
The presentation describes the process of pretesting online-questionnaires and the ‘(re)-search of basic requirements’. Background of applying this term is that, when defining a profile of requirements, it is linked to the basic experience, that even thoroughly selected design approaches by researchers may not meet users’ needs in practice. Through the eye of the respondent, pretests deliver insights into survey reality and therefore complement results from research literature. Taking results from pretests together with facts from research, lead us to conclusions on how to enhance the design of online-questionnaires.